Skip to content

Store provider credential overrides encrypted in the database - #492

Merged
jrhizor merged 10 commits into
mainfrom
jrhizor/db-secret-storage
Jul 27, 2026
Merged

Store provider credential overrides encrypted in the database#492
jrhizor merged 10 commits into
mainfrom
jrhizor/db-secret-storage

Conversation

@jrhizor

@jrhizor jrhizor commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Adds the storage layer for overriding provider credentials from inside the app, so a later PR can add the UI. It is additive: nothing is moved off environment variables here. With no stored secrets, every deployment behaves exactly as it does today.

How it works

Providers read credentials through getCredential(name) — a stored override if one exists, otherwise process.env. Overrides live in a secrets table as compact JWE, keyed by the environment variable each one stands in for (OPENAI_API_KEY, OXYLABS_PASSWORD, …), and are refreshed into an in-memory overlay every minute.

Keying by variable rather than by provider is deliberate: a provider isn't the unit anyone edits. BrightData needs a login and a password, and grouping them only forced a bundle protocol on top of what is really a per-variable override.

  • Uses jose with direct key management (dir) and A256GCM, so application code never touches an IV or a tag. jose is already in the tree at this exact version via better-auth, so this adds no new dependency.
  • Binds each ciphertext to its variable name in the authenticated JWE header, so a row can't be re-homed onto another variable by someone with database write access.
  • Pins the accepted algorithms on decrypt. A JWE names its own algorithms, so without an allowlist a row re-encrypted under a weaker one would be honoured.

Keys and rotation

ELMO_ENCRYPTION_KEY is generated by elmo init and backfilled on upgrade, and is now required in local mode — the mode the CLI provisions. The hosted modes are provisioned out of band and keep no store of their own, so they don't require it.

Each payload records the id of the key that encrypted it: a domain-separated SHA-256 of the key, truncated. That makes a key self-describing, so there is nothing to configure and nothing to keep in sync. A rotation is then a restart rather than a re-entry of every credential — put the previous key in the comma-separated ELMO_ENCRYPTION_KEY_OLD and it stays readable while new values are written under the new key. A row naming a key nobody holds reports that key rather than blaming the payload.

Re-wrapping stored rows under the current key — which is what lets you finally drop ELMO_ENCRYPTION_KEY_OLD without re-entering anything — belongs with the write path, so it lands with the UI.

Cloud

Nothing cloud-specific, and no deployment-mode branching anywhere in this path. Infisical secret syncs land provider credentials in the web and worker environments; ELMO_ENCRYPTION_KEY isn't set there, so getCredential reads process.env and the secrets table is never queried.

Failure behavior

Environment credentials are always the fallback, so nothing here can fail a startup:

  • A wrong-length key, a retired key set with no current one, or a row that won't decrypt logs an error naming the action to take and contributes nothing. One bad row can't take out the others.
  • A failed refresh keeps the last good overlay and logs an error.
  • The overlay is swapped only after every row has been read.

The worker awaits the first load so a stored credential counts toward SCRAPE_TARGETS validation. The web app leaves it running in the background rather than delaying boot.

Verification

  • pnpm test — 574 tests passed
  • Web, worker, and CLI typechecks passed
  • Deployment-mode smoke checks passed for local, demo, whitelabel, and cloud
  • Exercised against the real code path: JWE header shape, a fresh IV per encryption, algorithm-downgrade and cross-name replay rejection, and a full rotation — write under the old key, rotate, read, re-save, drop the old key

@vercel

vercel Bot commented Jul 23, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
elmo Ready Ready Preview, Comment Jul 27, 2026 8:13pm

Request Review

@jrhizor jrhizor changed the title Store provider credentials encrypted in the DB (dormant) Use JOSE-encrypted credentials locally and Infisical in cloud Jul 23, 2026
Base automatically changed from jrhizor/provider-spend-caps to main July 24, 2026 09:13
Fail loudly instead of silently unconfiguring providers. An Infisical read
that succeeds but returns nothing is an outage, not an empty credential set,
so the loader throws and the overlay keeps its last good values. Misread keys,
undecryptable rows and vanished credentials log errors with the action to take.

Take Infisical off the web request path. The cloud web app runs on Vercel
serverless, where a runtime read would add an authentication round trip to
every cold start and put Infisical in front of the whole site; Infisical's
Vercel secret sync populates the environment at deploy time instead. Only the
worker, which is long-lived and spends against these credentials, keeps the
SDK loader — so the Infisical vars are no longer web startup requirements.

Read one Infisical folder rather than recursing from the project root, so the
machine identity needs no broader grant and two folders cannot both define
OPENAI_API_KEY with no defined winner.

Drop what the payload does not need: typ and v leave the JWE header, since ctx
is what stops a ciphertext being re-homed onto another provider. Drop the hint
column, which stored the last four characters of the longest value in a
bundle — the password, for Oxylabs and DataForSEO — in plaintext beside the
ciphertext, and which nothing reads.

Delete the unused dataforseo client that captured credentials from the
environment at module load.
The worker loads credentials before pg-boss starts, so a source that throws
took the whole worker down. On an upgrade that source is a fresh query against
provider_credentials, which fails if the database is briefly unreachable or not
yet migrated — and the compose file only guarantees migration ordering when
Postgres runs in Docker.

Schedule the retry interval before the first load and let self-hosted modes
carry on with their .env credentials when that load fails. Managed cloud still
rethrows, since it has no environment fallback and a worker that started anyway
would only take jobs it cannot run. This also fixes the web path never retrying
after a failed initial load.
Only the worker reads them at runtime, but treating them as worker-only let a
cloud deployment go out with a worker that had no way to reach Infisical. Make
them cloud requirements again so that misconfiguration surfaces at startup.

INFISICAL_SECRET_PATH and INFISICAL_SITE_URL stay optional — both have working
defaults, and requiring the site URL would break any deployment relying on the
US cloud default.
@infisical/sdk depends on @aws-sdk/credential-providers and @smithy/*, and Nitro
traces a dynamic import into the output whether or not the branch can run. The
cloud web app never loads credentials over the network, so that was megabytes of
AWS SDK shipped into a Vercel serverless function that would never call it.

Move the worker's source selection into its own entry point. startCredentialRefresh
now takes a source and whether it is required, so the module the web app imports
only ever references instanceCredentialSource.

Tracing the import graph from apps/web/src/server.ts no longer reaches
@infisical/sdk; from apps/worker/src/index.ts it still does.
Both only ever held their defaults, and each was another way for a cloud
deployment to point the loader somewhere the credentials are not. Read the root
of INFISICAL_ENVIRONMENT on Infisical's US cloud, with no way to override
either.

With the read already non-recursive, that fixes the folder contract in one
place: one flat set of canonically named secrets at the environment root, and
nothing else in the project is fetched.
Infisical reaches both cloud runtimes through secret syncs, which land the
provider credentials in each platform's environment. The app needs no
Infisical client of its own: getCredential already falls through to
process.env, and with no ELMO_ENCRYPTION_KEY there is no store to consult,
so cloud is pure environment with no deployment-mode branching at all.

That removes the reason the refresh loop lived in @workspace/deployment, so
it moves to @workspace/lib/secrets alongside the store it drives.

Storage is keyed by the environment variable a secret overrides rather than
by provider, since a provider is not the unit anyone edits — brightdata
needs a login and a password, and grouping them only forced a bundle
protocol on top of what is really a per-variable override.
@jrhizor jrhizor changed the title Use JOSE-encrypted credentials locally and Infisical in cloud Store provider credential overrides encrypted in the database Jul 27, 2026
`elmo init` generates the key and the CLI backfills it on upgrade, so every
self-hosted deployment has one — validating it makes that guarantee visible
instead of leaving a deployment silently unable to store credentials. The
hosted modes are provisioned out of band and keep no store of their own, so
the requirement is scoped to local.
A JWE names its own algorithms, so the allowlist on decrypt is what stops a
row re-encrypted under a weaker one from being honoured. Nothing covered it.

Also retires the provider-shaped AAD strings the crypto tests still used.
Without a key id there was no rotation path: a payload could not say which
key it belonged to, so the store could not tell a rotated key from a corrupt
row, and changing the key stranded every existing secret with no way back.

ELMO_ENCRYPTION_KEY still encrypts, and the comma-separated
ELMO_ENCRYPTION_KEY_OLD stays readable, so a rotation is a restart rather
than a re-entry of every credential. A row naming a key nobody holds now
reports that key instead of blaming the payload.

The id is a domain-separated SHA-256 of the key, truncated — self-describing,
so a key identifies itself with nothing to configure and nothing to keep in
sync. Selecting on it is safe: the header is the AEAD's additional data, so a
rewritten kid fails the tag rather than steering the payload somewhere it
decrypts.
@jrhizor
jrhizor merged commit c2ae13a into main Jul 27, 2026
9 checks passed
@jrhizor
jrhizor deleted the jrhizor/db-secret-storage branch July 27, 2026 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant